Chapter 19 Kolmogorov Complexity

Kolmogorov complexity has intellectual roots in the areas of information theory, computability theory and probability theory. Despites its remarkably simple basis, it has some striking applications in Complexity Theory. The subject was developed by the Russian mathematician Andrei N. Kolmogorov (1903–1987) as an approach to the notion of random sequences and to provide an algorithmic approach to information theory [1]. A similar theory was described by Ray J. Solomonoff (1926–) in his study of inductive inference [5, 6]. For a comprehensive treatment of this subject, including the history of this subject, we refer to the excellent book of Li and Vitányi [4]. We give some intuitions that point to Kolmogorov complexity. Consider the following binary strings: (The commas are just a visual aid for parsing the strings). If these strings are being continued, we might ask whether the next bit can be predicted based on the first 16 bits shown. For instance, a natural guess is that the 17th bit of x i is i for i = 0, 1. It is less obvious 1 what ought to be the next bit in x 2. This question leads to predictive theories about strings from examples, or more generally, the problem of inductive inference. We can also ask for the " shortest descriptions " of strings such these. Thus x 0 might be described as " a string of zeros of lenghth 12 ". If x 0 is considered an infinite string, we want the shortest description of an extension of the first 12 bits. This might be " a string of all 0's ". If K(x) is the length of the shortest description of x (within some suitable formalism), then we can interprete K(x) as the " information content " of x. One can further view those strings x where K(x) is approximately |x| to be " random ". This makes connection with the theory of random strings. Kolmogorov complexity is essentially the study of the function K(x). In the rest of this introduction, we provide some background concepts and unifying notations. Bit Strings and Natural Numbers. Unless otherwise noted, " strings " shall mean bit strings and " numbers " shall mean natural numbers. We interchangeably view a bit string x ∈ {0, 1} * as a number. Let denote the empty string, and |x| denote the length of x. Note that xy can mean concatenation (if x, y …